Sleep or physical activity?

I would ideally love to maintain a consistent exercise routine. However, there are stretches of time during which I get very little sleep, due either to a hectic schedule or a lot of stress. During these times (sometimes one or two weeks), I find it almost impossible to go work out. I'm simply too fatigued (I don't drink caffeine because it disrupts my sleep ? even one cup in the morning!). The result is that I start to get flabby and untoned, and I then I tend to fluctuate between almost-toned to back-to-flabby.